Feature/GitHub ingest - #2
Merged
Carr1005 merged 3 commits intoJul 7, 2026
Merged
Conversation
Automate producing the notebooks "context" markdown by cloning a course repo over SSH instead of hand-downloading it, and — new — include the repo's helper.py module(s) as context. - New .claude/skills/generate-spec/scripts/ingest_repo.py (stdlib only): clones a repo (SSH/https/local), renders every .ipynb cell-by-cell in the existing context.md format under a Lesson Map, and appends a de-duplicated "Helper Module Context" section. Helper defs (functions and classes) are extracted via ast; identical/symlinked copies collapse; same-name-but-different-body collisions keep both, flagged inline. Output is guarded to materials/notebooks/ and never touches transcripts/. - generate-spec SKILL.md: additive optional Step 2 — a repo argument triggers ingestion; no argument behaves exactly as before. Transcripts remain a manual drop. - new-course SKILL.md: mention the /generate-spec <repo-ssh-url> shortcut. - .gitignore: ignore .idea/ and Python bytecode. Verified end-to-end against a local fixture (symlink collapse, conflict flagging, zero-helper, clone failure, output guards) and against the real course repo (5 notebooks, helper.py surfacing 22 deduped defs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cover the new /generate-spec repo argument: accepted sources (SSH, HTTPS, local path/file://), what ingest_repo.py renders, and its scope guarantees (notebooks-only writes, manual transcripts, idempotent).
bluegreenstone
pushed a commit
to bluegreenstone/spec-build-lab
that referenced
this pull request
Jul 27, 2026
PR https-deeplearning-ai#2 bolted an optional repo-ingestion step onto /generate-spec. The capability is useful; the placement was not. It forced generate-spec to carve out an exception in its own "Don't write to materials/" rule, and made a single-purpose skill do two jobs. Every other skill in the loop is single-purpose and hands off. This extracts it rather than reverting it: the script moves with its history (git mv), the prose moves largely intact, and the shortcut keeps working behind its own front door. /generate-spec is materials -> spec.md again, with a flat write invariant; /ingest-course-repo is the sole writer into materials/. Two bugs fixed on the way: - Silent clobber. The prescribed --out was byte-identical to the name of a hand-downloaded context dump (<course>-context.md), and the script overwrites without asking. Output now lands in materials/notebooks/from-repo/, so the two coexist. - The target repo was un-ingestible. deeplearningai-eng/courses is a 3.4 GB monorepo holding ~260 courses with no path scoping in the script, so ingesting it would have cloned everything and merged every course into one context file. ingest_repo.py gains an optional --subdir that scopes discovery and switches the clone to --filter=blob:none --sparse with GIT_LFS_SKIP_SMUDGE=1. Pulling course_10 now transfers 8.7 MB in ~8s. Omitted, behaviour is byte-identical to before, so standalone course repos are unaffected. The skill parses a GitHub tree URL into repo + --ref + --subdir, and refuses the bare monorepo root with a pointer to browse for the course folder. Framing throughout is corrected from PR https-deeplearning-ai#2's "skip the manual notebook download" to what it actually is: an optional supplement that adds the course's real code and helper.py on top of the site download, which remains the main path. Verified: backward compatibility against the pre-change script on a local fixture (byte-identical); subdir scoping; sparse clone size; bad-subdir and transcripts-path guards; no clobber of a hand-downloaded file; tree-URL parse producing the same output as hand-typed flags; both refusal paths firing without invoking git. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add optional git-repo notebook ingestion to /generate-spec
/generate-spec now accepts an optional git repo argument (/generate-spec ). When given one, it runs a deterministic ingest step before generating the spec, so notebooks no longer have to be downloaded and copied in by hand.
What the ingest does (.claude/skills/generate-spec/scripts/ingest_repo.py, stdlib-only):
Guardrails:
Also: